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ABSTRACT 

This paper argues that college administrators have largely 
escaped the scrutiny of the otherwise far-reaching and comprehensive outcomes 
assessment movement in higher education. This paucity of assessment data 
regarding managers in higher education is especially remarkable in light of 
the fact that many internal and external constituency groups generally view 
college administrators, not faculty, as being responsible for initiating 
improvements in institutional performance. The author examines current trends 
and best practices in the assessment of academic administrators, draws 
comparisons between administrative ef f ectiveness measures and those used for 
curricular and institutional effectiveness measures, and makes 
recommendations for improving measures and methods of administrative 
effectiveness. The author distinguishes between the terms "accountability" 
and "responsibility" as they are used in higher education. Both the public 
and the political bodies that represent the public view administrative 
leaders as accountable for the performance of higher education institutions, 
while higher education administrators view faculty as responsible for 
curricular responsiveness and change. One hypothesis concerning the current 
status of assessment measures involves the possibility that higher education 
leaders may have responded too quickly to the public cry for accountability, 
perhaps missing the fundamental questions being asked as a result. Contains 
40 references. (AUTH/NB) 
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Abstract 

College administrators have largely escaped the scrutiny of the otherwise far-reaching 
and comprehensive outcomes assessment movement in higher education (Heck, Johnsrud, & 
Rosser, 2000; Hoekema, 1999). This relative paucity of assessing managers in higher education 
is remarkable in light of the observation that many internal and external constituency groups 
generally view college administrators, and not the faculty, as responsible for initiating 
improvements in institutional performance (Banta, 1993; Bimbaum, 2000; Ewell, 1993; 

Gumport & Pusser, 1997; McLaughlin, 1996; Seldin, 1988; Young & Knight, 1993). The 
purpose of the present article is to examine current trends and best practices in the assessment of 
academic administrators. Comparisons are drawn between administrative effectiveness 
measures, and those used for curricular and institutional effectiveness procedures. Finally, 
recommendations for improving measures and methods of administrative effectiveness are 
proposed. 
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Assessment of Administrative Effectiveness 

College administrators have largely escaped the scrutiny of the otherwise far-reaching 
and comprehensive outcomes assessment movement in higher education (Heck, Johnsrud, & 
Rosser, 2000; Hoekema, 1999). Despite the ever-increasing interest of state legislators, 
governing boards, accrediting agencies, business leaders and the general public regarding higher 
education accountability (Atwell, 1996; “AGB’s top ten policy issues”, n.d.; Banta, 1993; 
Diamond, 1998; Gibbs, 1999; Guskin & Marcy, 2001; Waller, Coble, Scharer, & Giamportone, 
2000), relatively little emphasis has been placed on consistently measuring and improving 
administrative effectiveness in higher education (Dyer & Miller, 1999; Heck, et al., 2000; 
Hoekema, 1999; Scarborough, 1999; Seldin, 1988). The relative lack of effort in assessing 
managers in colleges and universities is remarkable in light of the observation that many internal 
and external constituency groups generally view administrators, and not the faculty, as 
responsible for initiating improvements in institutional performance (Banta, 1993; Bimbaum, 
2000; Ewell, 1993; Gumport & Pusser, 1997; McLaughlin, 1996; Schmidt, 2002; Seldin, 1988; 
Young & Knight, 1993). 

College administrators have, of course, been evaluated in higher education for many 
years (Seldin, 1988). However, many of these evaluative processes differ in purpose and scope 
from the spirit of the outcomes assessment and institutional effectiveness movements that have 
pervaded the higher education landscape of the past two decades. Evaluations are typically 
viewed as a necessary method of providing a supervisor with information relative to one specific 
employee, and for the purpose of making personnel-related decisions relative to that employee. 
In other words, evaluations provide quantifiable data for “high stakes” personnel decisions, such 
as promotion, dismissal, and contract renewal (Heck, et al., 2000; Seldin, 1988). Nonetheless, 
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Seldin (1988) promoted the broader perspective that evaluations ought to be conducted 
fundamentally to identify an employee’s need for improvement, to target weaknesses so as to 
recognize professional development needs, and ultimately to enhance institutional performance. 

Seldin’s (1988) point of view notwithstanding, evaluation methods are often ineffective 
for promoting positive institutional change because they are rooted in politics (Heck, et al., 
2000), lack benchmark comparisons with similar positions across higher education (Dyer & 
Miller, 1999) or fail to meet many of the fundamental requirements of successful appraisal 
systems (Seldin, 1988, p. 40). As an aside, a story told by Masiello (1999) about Woodrow 
Wilson highlights the fact that politics is not a new topic in higher education. Masiello writes 
that when Wilson left the presidency of Princeton University for a position in government, he 
quipped “he had learned the art of politics from the professionals and was now going to 
Washington to practice with the amateurs” (Masiello, 1999, p. 61). 

These issues of politics and subjectivity in administrative evaluation are not unlike 
similar concerns faced by practitioners of the institutional effectiveness and assessment 
movements. These movements have gained prominence in higher education as a direct reaction 
to widespread public dissatisfaction about the purpose and effectiveness of higher learning, and 
are accompanied by a strident call for accountability on the part of policymakers (Bimbaum, 
2000, Lucus, 1996). Resource constraints contribute to higher education’s thrust toward quality 
and efficiency, according to Gumport and Pusser (1997), and the end goal of such efforts is a 
drastic re-conceptualization of higher education institutions. Despite this propulsion of higher 
education toward the requirement of measuring effectiveness, the assessment movement has 
been characterized by conflict over the identification of quality standards, the operational 
definitions of such standards, attempts to quantify the values inherent in the teaching/leaming 
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process, and the uneven application of criteria relative to quality measures (Lucas, 1996; Miller, 
Finley, & Vanko, 2000). 

The purpose of the present article is to examine current trends and best practices in the 
assessment of academic administrators. Comparisons are drawn between administrative 
effectiveness measures, and those used for curricular and institutional effectiveness procedures. 
Finally, recommendations for improving measures and methods of administrative effectiveness 
are proposed. 

The Call to Accountability 

Ironically, accountability and responsibility are frequently not aligned in institutions of 
higher learning. For example, Ferren (1997) faults the delegation of curricular authority from 
boards and presidents to the faculty as contributing to the confusion between college 
responsibility and fiscal accountability. Specifically, she views the lack of contact between the 
faculty responsible for the development of courses and curricula, and the administrators 
accountable for cost containment and fiscal stability, as an obstacle to the nearly impossible task 
of managing the curriculum for fiscal stability (Ferren, 1997, p. 534). 

This disconnect between responsibility and accountability is a unique characteristic of higher 
education that derives directly from the historical form and function of the enterprise, an enterprise 
in which decision making is diffused throughout the organizational structure (Bimbaum, 1988). 

This unique structure is even more problematic in that there is general disagreement among the 
various constituency groups within higher education (e.g., faculty, administrators, alumni, staff and 
students) regarding the amount of influence desired from the very source of legal authority for 
collegiate decision making, boards of trustees (Bimbaum, 1988). Finally, this ambiguity is further 
complicated by the fact that many collegiate boards of trustees are rejecting the notion that they 
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should function only as policy overseers and are taking a more active role in everyday governance 
decisions (Fisher & Koch, 1996; Ingram, 1996; Seldin, 1988). 

One reason why colleges have struggled so much with the accountability mandate, and 
with the changing nature of governmental roles relative to postsecondary education (Gibbs, 

1999), is because the unique historical attribute of diffused decision making in higher education 
is in direct conflict with public perception. The public believes that those responsible for 
curricular changes are also accountable for their outcomes, mainly because that is the way most 
businesses function. Arguably both the public, and the political bodies that represent the public, 
view administrative leaders as accountable for the performance of higher education institutions. 
Meanwhile, higher education administrators view faculty as responsible for curricular 
responsiveness and change. The public finds it unfathomable that change cannot occur in higher 
education through administrative fiat. 

This clash between accountability and responsibility is described by Dill (1982) as a 

conflict between ideology and bureaucracy within academic institutions, a quarrel that always 

results with ideology as the victor. Dill (1982, p. 264) writes: 

The very authority of the enterprise therefore rests on obedience to a set of values or 
ideological norms. For example the legitimacy of rules and regulations within academic 
institutions is determined by their consistency with the goals of academic ideology. 

When professional commitment to these norms (e.g., the pursuit of truth), comes into 
conflict with bureaucratic rules (e.g. governmental requirements that researchers obtain 
the consent of their subjects) priority is given to ideology (e.g. many researchers ignoring 
the regulations in conducting the research). 

Dill’s use of the research example could be easily replaced with the governmental call to 
accountability. Perhaps the bureaucratic notion of measuring the outcomes of learning clashes so 
fundamentally with the unquestioned ideology of the teaching enterprise, that assessment 
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regulations are easily dismissed by faculty because of their shared belief about the value of 
instruction. 

In the same fashion, shared ideology among academic administrators concerning their 
roles and functions may contribute to difficulty in assessing their effectiveness. Many of the 
problems of measurement that plague other accountability movements in higher education also 
are relevant to administrative evaluation. Concern over the fairness of subjective evaluation, the 
inability to create measurable outcome statements related to job performance, and the variety of 
roles assigned to academic administrators, even those with similar job titles, all contribute to 
measurement difficulty. 

Ironically, many of the fears about administrative evaluation that are expressed by 
administrators in higher education are indistinguishable from the arguments faculty members use 
when criticizing measures of teaching effectiveness. Few individuals like their performance to be 
judged (Miller, et al., 2000), especially when such judgments are made subjectively (Heck, et al. 
2000), or on the basis of the relative achievements of others, i.e. student performance (Muffo, 1996). 
Despite these concerns, faculty and administrators will have little choice but to “tolerate this kind of 
assessment measurement” despite the tacit recognition that “the efficacy and quality of faculty work 
involve imponderables not reducible to any meaningful quantification” (Lucus, 1996, p. 239). Eaton 
(2001) emphasizes the enduring nature of the contradiction between the inevitability of 
accountability standards on the one hand, and the inadequate measures of accountability on the 
other. Despite the view of some in academe that the focus on student outcomes “is a call to reduce 
the teaching and learning experience to measurable objectives that cannot capture the true richness 
of the collegiate experience”, critics of postsecondary education contend that current methods of 
learning outcomes clearly will not satisfy all of higher education’s constituents (Eaton, 2001, p. 43). 
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Historically, higher education administrators have viewed resource acquisition, 
allocation, and spending as synonymous with quality (Lucus, 1996). However, as the public 
demanded more tangible measures of institutional success at the collegiate level, student learning 
outcomes became the focus of institutional effectiveness measures toward the goal of 
documenting that graduates have actually accomplished “learning” (Lucus, 1996). Chaffee 
(1998) describes this response to external demands for.accountability as higher education’s new 
“customer focus”. However, Chaffee also cites the difficulties associated with achieving a 
common understanding of this term as the single greatest factor contributing to the declining 
quality of management in colleges. 

One intriguing hypothesis concerning the current status of higher education assessment 
measures involves the possibility that, perhaps defensively, higher education leaders may have 
responded too quickly to the urgent public cry for accountability. In their haste to provide some 
reasonable response to questions about the fundamental purpose and nature of higher education, 
questions rarely asked in previous generations, administrators responded by providing a tidal 
wave of quantifiable data. In the rush to justify all that is done in higher education, perhaps we 
have missed the fundamental questions truly being asked by an impatient public. 

Chaffee (1998) implies this point by suggesting that answers provided to the public by 
higher education leaders thus far, have merely generated more questions. While sufficiently 
providing the public with plenty of data about faculty workload, for example, Chaffee points out 
that we have failed to really hear and understand the very question asked of us by a skeptical 
public. Asserting that the public simply wants to see the educational benefits resulting from the 
increasing costs associated with higher learning, Chaffee advocates the adoption of an attitude of 
caring toward students as a method of providing student “customers” with those benefits. 
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Chaffee’s (1998) “attitude of caring” may prove helpful in improving the assessment 
process itself. There are two steps to providing the public with the answer to their questions 
concerning the benefits of higher education. First, all of the internal constituency groups in 
higher education must assume an attitude of caring toward the unquestioned ideology that 
defines the enterprise. In other words, college and university leaders, faculty and administrators 
alike, must articulate more forcefully and believably the intangible benefits associated with 
undergraduate education. Most of these benefits derive from ideologies concerning liberal 
education, and organizations like the American Association of Colleges and Universities have 
recently recognized the need to proclaim these benefits more directly to the general public 
(American Association of Colleges and Universities [AAC&U], 2002). Secondly, assuming 
widespread belief in the ideology of higher education benefits is possible, higher education 
leaders must, to the best of their ability, operationalize and measure these benefits. This ought to 
be done even while simultaneously promoting the belief that the sum of such measurements fails 
to describe the overall beneficial experience associated with undergraduate education. The 
practice of measuring the benefits of higher learning involves the assessment of both institutional 
effectiveness and administrative effectiveness programs toward the common goal of improving 
student learning in academe. 

Comparisons between Institutional and Administrative Effectiveness Programs 

Ronco and Brown (2000) list four elements of effective institutional assessment 
programs. They are the critical assessment of student learning, the articulation of program goals, 
the setting of standards for accomplishment, and the measurement of the extent to which such 
standards are met. The ultimate goal of institutional effectiveness procedures is to demonstrate 
that higher education makes a difference (Banta, 1993), and to move an institution toward the 
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continuous improvement of program outcomes (Chapin, 1999). Seldin (1988) points out that the 
practice of administrative evaluation is geared toward the identification of an employee’s need 
for improvement in order to target weaknesses and recognize professional development needs. 
However, Seldin further argues that the end result of administrative evaluation procedures is, as 
it is the case in institutional assessment, the enhancement of institutional performance. 

Therefore, both institutional effectiveness and administrative evaluation programs can be 
viewed as two sides of the same coin. Both processes are designed to accomplish the same end: 
institutional improvement based on student outcomes. Furthermore, both processes provide data 
that responds, in part, to external cries for measures of accountability. Despite these between 
institutional and administrative effectiveness procedures, Seldin suggested in 1988 that 
administrative evaluation processes lag well behind institutional effectiveness programs in 
methodological sophistication (Seldin, 1988). Not much has improved in administrative 
evaluation methodology since the late 1980s, and my contention is that this is principally due to 
the remarkable disregard of administrative accountability by higher education leaders. 

Since the outcomes assessment and administrative evaluation movements share similar 
goals, the latter would do well to examine the successful practices of institutional effectiveness 
programs in order to close the methodological gap between the two. Ronco and Brown’s (2000) 
four elements of effective institutional assessment programs provide a reasonable starting point 
from which to examine administrative evaluation methods. The first element, the critical 
assessment of student learning, constitutes the fundamental task of effective assessment 
programs, measuring standards related to program objectives (Chapin, 1999; James Madison 
University, 1998). Outstanding administrative evaluation programs similarly require a unifying 
task, the measurement of administrative standards relative to institutional objectives. The second 
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element of effective institutional assessment programs involves the articulation of program goals, 
a task that is paralleled on the administrative assessment side by the articulation of administrative 
goals. The third element, the setting of standards for accomplishment, is much more 
complicated in administrative evaluation. That is due to the fact that four different types of 
standards exist in personnel related assessment (Heck, et al. 2000). These four types of 
standards, first described by the Joint Committee on Standards for Education Evaluation (as cited 
by Heck, et al., 2000), make it difficult for faculty, staff and administrators to agree on which 
accomplishments administrative evaluation standards will focus on. Stufflebeam and Nevo (as 
cited in Heck, et al., 2000, p. 666) define the four types of standards. Utility standards involve 
the aspects of evaluation that ensure informative, timely, and influential products. Proprietary 
standards are those associated with ethical constraints that ensure the rights of all are protected in 
the process of evaluation. Feasibility standards refer to the efficiency and ease of the evaluation 
process. Finally, accuracy standards infer the dependability of the information concerning 
administrative performance. Heck, et al. (2000) demonstrate that the various constituency 
groups in higher education (deans, faculty and staff) pay attention to different standards while 
developing criteria for the administrative evaluation process. 

As a consequence of the variety of evaluation standards in personnel assessment, the 
fourth element of institutional effectiveness, the measurement of standards, also becomes 
clouded when applied to administrative evaluation procedures. All four of elements constitute 
important features of both institutional and administrative effectiveness programs. Therefore, a 
more detailed analysis of how each of them might assist practitioners to develop an effective 
administrative evaluation process follows. 
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Applying the Elements of Institutional Effectiveness to Administrative Evaluation 

Element one: Defining the unifying outcome associated with administrative effectiveness. 

The link between institutional effectiveness and measured student learning appears to 
have face validity. An institution of higher learning is only as effective as the documented 
learning growth observed in its students. When examining administrative effectiveness however, 
Seldin (1988) provided a useful distinction between measures of outcomes and measures of 
behaviors. Seldin points out that academic administrators have traditionally been judged by 
measures of behavior. Conversely, the institutional effectiveness movement has traditionally 
emphasized outcome measures. Outcome orientations emphasize result measurements, whereas 
behavior orientations emphasize activity measures (Seldin, 1988). 

Measuring administrative effectiveness by what an administrator does is problematic, 
mainly because of the myriad of differing roles that administrators play (Dyer & Miller, 1999; 
Gmelch, Wolverton, Wolverton, & Sarros, 1999; Heck, et al., 2000; Hoekema, 1999; Tucker, 
1992), because those roles continue to change over time (Seldin, 1988), and because such roles 
are subject to the prevailing political motivations of the time (Heck, et al., 2000). Townsend and 
Bassoppo-Moyo (1997) add further fuel to the complexity of roles fire by asserting that few 
Studies have actually asked college administrators about the work they do, what they need to 
know, or what characteristics are important for academic leaders to possess. These authors 
undertook that task by evaluating the data from an open-ended survey on essential knowledge 
and skills needed by current academic administrators in community colleges. The researchers 
looked at responses from 61 administrators (out of 160 surveys distributed) and found that many 
administrators felt inadequately prepared for making budgetary, legal and personnel decisions. 
The authors concluded that higher education administration graduate programs ought to require 
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coursework in finance, law and budgeting, as well as courses that emphasize competence in 
interpersonal relations. 

One way to avoid the problems associated with the complexity of roles inherent in 
administrative positions is by emphasizing particular aspects of the academic manager’s role 
over others in the evaluation process (Heck, et al., 2000). However, Heck et al (2000) caution 
that the criteria resulting from the delineation of administrative roles should be grounded in 
research and practice, and well articulated in the administrative job description. 

In summary, to achieve the clear description of outcomes for administrative effectiveness 
measures, evaluation procedures must move away from behavioral standards and toward a 
results orientation. Such an outcomes focus must take into account the myriad of roles 
incumbent upon academic administrators. Evaluation methods must include advance criteria, 
written into job descriptions, upon which effectiveness will be measured. Ideally, the one 
unifying outcome for measuring administrative effectiveness should be the same outcome for 
judging institutional effectiveness. That is the answer to the question regarding the extent to 
which the administrator can claim credit for moving the institution toward the goal of improved 

l 

student learning. Unlike faculty however, the administrator must also be concerned with the 
conflicting role of achieving this outcome within the limits of budgetary constraints (Dyer & 
Miller, 1999). 

Element two: Articulation of administrative goals. 

The second element of institutional effectiveness involves articulation of the goals that 
derive from the outcome of student learning (Ronco & Brown, 2000). In other words, college 
and university leaders are called upon to articulate, in advance, how they strive to achieve the 
end result of learning. When applied to administrative effectiveness, the task that emerges is to 
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express the manner in which administrative oversight will contribute to student learning within 
the context of cost containment. Hammons & Murry, Jr. (1998) argue that academic 
administrators are well equipped to perform the academic functions, but they are ill equipped to 
perform the managerial/fiscal functions. 

Employing the analogy of the financial audit, Hammons & Murry, Jr. (1998) advocate the 
use of a “Management Audit Questionnaire” (MAQ), developed by Murry, Jr. (1994), to assist 
administrators in determining their managerial effectiveness. Bemoaning the fact that most 
administrators are hired based merely on their educational credential and past work experience, 
the authors assert that knowledge of, or experience with, managerial principles are woefully 
lacking in collegiate administrators. As a result, the authors believe that administrators are not 
well suited to perform the necessary tasks associated with the functions and roles of a manager. 

A key benefit to employing the MAQ for managerial assessment is that both managerial 
and academic criteria can be measured because of the flexibility of the instrument. Hammons & 
Murry, Jr. (1998) emphasize the use of the questionnaire as part of an overall process of planned 
change. The items on the MAQ associated with managerial assessment were developed after 
extensive literature review and were subject to intensive revision through a Delphi study 
technique with community college presidents. In addition to those criteria, a college-specific 
managerial assessment committee is free to develop further items that measure other aspects of 
administrative effectiveness. Once developed, the MAQ is administered to a wide range of 
individuals with knowledge about the performance of the administrator. After analysis of the 
data, an individual appraisal of the manager’s skills can be provided, and professional 
development can address areas of needed improvement. 
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In summary, academic administrators must set goals that achieve the sometimes 
contradictory objectives of academic excellence and fiscal constraint. The analogy of a fiscal 
audit applied to managerial qualities provides a useful framework for administrators to assess 
their ability in these two conflicting areas. Specifically, the Management Audit Questionnaire 
(Hammons & Murry, Jr., 1998; Murry, Jr., 1994) provides a needed tool for the assessment of 
administrative effectiveness relative to the goals of student learning and fiscal constraint. 

Element three: Setting standards. 

The third element of institutional effectiveness involves setting standards by which the 
implementation of goals may be judged (Ronco and Brown, 2000). In order to develop criteria 
for administrative effectiveness, the standards employed for such criteria must be well 
understood. Heck, et al (2000) argue that the most useful criteria evolves from standards when 
specific information desired is matched with the sources most qualified to provide such 
knowledge. However, greater understanding of the various types of standards that can be 
employed (utility, proprietary, feasibility and accuracy) needs to precede determination of the 
criteria employed to judge effectiveness. Furthermore, as stated previously, specific information 
desired for administrative effectiveness emerges from a clear definition of the roles expected of 
administrators. Academic administrators report that job satisfaction is negatively correlated 
with ambiguity of administrative roles (Gmelch, et al., 1999; Heck, et al., 2000; Wolverton, 
Wolverton & Gmelch, 1999). Dyer & Miller (1999) contend that role complexity has contributed 
to the difficulty in advancing the literature on the topic of administrative assessment. 

The myriad of roles expected of academic administrators contributes to the difficulty 
associated with developing assessment standards related to administrative effectiveness. The 
information desired from administrative evaluation emerges from the clear delineation of roles 
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for each administrator. Determining the specific information is desired from administrative 
evaluations, along with the persons most qualified to provide such information, are critical tasks 
in the establishment of administrative effectiveness programs. 

Element four: Measuring standards. 

Perhaps the most important step in the process of determining the effectiveness of an 
institution is to measure the standards by which the college has asked to be judged. However, 
this has also been demonstrated to be the most difficult, in that many in academe resist the 
reduction of the educational enterprise to a set of measurable objectives (Eaton, 2001; Lucus, 
1996). The same resistance is characteristic of administrative evaluation, as I have previously 
pointed out. Furthermore, as Heck, et al (2000) advise, administrative evaluation involves the 
subjective perception of effectiveness, a characteristic that lends itself to political, reprisal, and 
control motivations. 

Ironically, self-evaluation may be subject to personal bias as well. Kruger & Dunning 
(1999) offer evidence that the most incompetent people tend to hold overly favorable views 
regarding their own abilities. The authors further suggest that this over-inflation of cognitive and 
social competency by the lowest quartile of achievers occurs partially as a result of 
metacognitive deficiency. In other words, the most incompetent people fail to perceive the level 
of their own incompetence. 

As a result of these concerns involving subjective bias in administrative evaluation, a 
critical element of measurement must center on the psychometric properties of the assessment 
instrument used. In other words, the reliability and validity of the evaluation tool ought to be a 
focus of concern on the part of the evaluator (Heck, et al., 2000; Hammons & Murry, Jr., 1998; 
London, 1995). 
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Essential Elements of Administrative Effectiveness. 

Seldin (1988) listed seven essential elements of effective administrative appraisal 
systems. Consideration of these characteristics can assist evaluators in the development of 
appraisal systems that include fair and reliable measures of effectiveness. Relevance refers to the 
requirement to ensure that clear links exist between institutional goals and administrative 
performance standards. Seldin proposes that the relevance element can be determined on the 
basis of answering the question, “What really makes a difference between success and failure in 
a particular position?” (Seldin, 1988, p. 40). A second essential element of administrative 
appraisal programs is sensitivity. This characteristic refers to the ability of the system to 
distinguish poor performers from excellent ones. Reliability refers to the consistency of 
judgment across various appraisers, such that bias is eliminated to the greatest extent possible. 
Freedom from contamination is the element that ensures administrative appraisal is free from 
judgments about factors beyond the control of the person being evaluated. On the other hand, 
freedom from deficiency ensures that the appraisal system includes all of the relevant aspects of 
the administrator’s performance, even those that require inordinate amounts of time to collect or 
review. Practicality is the sixth essential element of effective appraisal systems suggested by 
Seldin. This refers to the ease with which the appraisal system can be understood and employed 
in the specific higher education environment where it is administered. Finally, Seldin asserts that 
acceptability is the most essential of all the elements. This element refers to the extent to which 
evaluators and those being evaluated trust the information provided by the appraisal system. 
Without acceptability, Seldin contends that appraisal systems will easily be subverted and 
destroyed. Improving the psychometric properties of evaluation systems can improve their 
acceptability by demonstrating the reliability and validity of the data collected in the process. 
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Current best practices in Administrative Effectiveness 

A few researchers have attempted to address the psychometric concerns associated with 

administrative evaluation. While advocating a greater role in collegiate decision making for 

human resource professionals, London (1995) promoted the use of “ratingless” evaluation 

systems. Decrying the use of ratings as merely a method of categorizing people, London 

believed that the elimination of rankings would promote a greater dialogue between the evaluator 

and the person evaluated. London (1995, p. 199) wrote: 

The goal of the new process is to move away from categorizing people toward providing 
more detailed summaries of performance linked to specific objectives. The ratings (even 
without forced choice) are a crutch that is highly subjective, likely to be inconsistent 
across groups (e.g. “exceeding objectives” may mean different things to different 
managers), and not necessarily backed by solid and well-documented performance 
information. Encouraging the meaningful review without the rating will focus attention 
on performance rather than distract attention to a subjective categorization. 

A second exemplary model for improving administrative effectiveness entails 

emphasizing the use of professional development over evaluation techniques. Dyer and Miller 

(1999) promote this perspective by discussing the woeful lack of department chairperson training 

programs that emphasize measurable outcomes. While recognizing the complex tasks associated 

with academic administration, the authors contend that the literature has failed to keep pace with 

demarcation of the best practices and benchmarking of administrative effectiveness. This is 

particularly important in light of the complexity of roles required of administrative positions, and 

Dyer and Miller cite professional development as the best method for managers to acquire the 

knowledge needed for excellent performance of those multiple roles. 

A third method of addressing psychometric measurement problems associated with 

administrative evaluation concerns the use of financial auditing techniques as a metaphor for 
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administrative evaluation. This “managerial auditing” technique, and the use of the management 
audit questionnaire (MAQ) developed by Murry, Jr. (1994), was discussed in a previous section 
of this article. Advocates of this auditing technique assert that its use will allow managers to 
gain a better understanding of the breadth of their tasks and organizations to gain a better 
understanding of the strengths and weaknesses of the management team (Hammons & Murry, 

Jr., (1998). Finally, these authors argue that the management audit technique provides more 
constructive feedback on the abilities of managers than traditional evaluation techniques are able 
to provide. 

The fourth and final method for addressing psychometric properties of administrative 
evaluation was proposed by Heck, et al. (2000). They advocate an innovative technique for 
reducing the bias associated with subjective evaluation. Conceding that evaluative appraisal is 
necessarily a subjective measure of perceptions, the authors promote the use of a statistical 
technique to ensure that assessment of those perceptions is reasonably accurate. Generalizability 
theory examines the multiple sources of error variance associated with subjective measurement 
techniques. 

The procedure described by Heck, et al (2000) began with the simple administration of a 
written questionnaire, which contains items that reflect the tasks, duties, and responsibilities of 
the administrative role. The instrument was initially administered to 229 faculty and staff who 
had relevant information and opinions concerning the leadership effectiveness of six particular 
deans. For each item on the questionnaire, a simple Likert scale ranking system was employed 
for evaluators to rank the leadership effectiveness of the deans. Initial statistical evaluation 
demonstrated each of the leadership subscales to be reliable. 
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Generalizability theory employs statistical techniques to differentiate variance related to 
the object of measurement (leadership ability) from variance associated with measurement error. 
Heck, et al. (2000) discount the crossed design procedure as impractical, but advocate the use of 
a nested design technique. This latter technique used a procedure wherein evaluators rated only 
their own dean and not the five other deans who were part of the study. 

Heck, et al. (2000) assert that in both designs, the error variance is evaluated relative to 
the desired information the evaluator wants to gain from the procedure. Relative error is helpful 
if the evaluator needs to make decisions about performance relative to the performance of other 
administrators. Absolute error assists the evaluator in making decisions concerning one 
particular dean’s individual scores. This latter type of error variance assesses the extent to which 
an administrator can capably perform the essential duties of the position. After initial evaluation 
of such error variance, subsequent studies (called D-studies for the resulting decisions made after 
their analysis) help the evaluator improve the quality of the evaluation process. For example, 

Heck, et al provide evidence from a D-study that supported their decision to increase the sample 
size of raters of one particular dean. Specifically, their decision to increase evaluators from 10 to 
30 vastly improved the reliability of the seven-item subscale they used to determine leadership 
effectiveness. 

To summarize, Heck et al. (2000) provide substantiation for the notion that Generalizability 
statistical theory can employed to easily provide accurate information about the effectiveness of 
administrators. The authors claim that virtually no other evaluation methods discussed in the 
literature provide evaluators with sufficient confidence to employ them for high stakes decision- 
making. Nonetheless, the most revealing result associated with the specific study conducted by 
Heck, et al was that only 1 1 percent of the variance associated with leadership rankings could be 
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attributed to true differences in leadership effectiveness. As the authors point out, “although there 
are real and measurable differences in the effectiveness of individual deans, there are even greater 
differences among the perceptions of effectiveness held by the faculty and staff rating the deans” 
(Heck, et al., 2000, p. 680). 

Toward an Eclectic Model for Administrative Effectiveness 

No one procedure described above adequately addresses all of the concerns associated with 
the subjective evaluation of academic administrators. However, each approach offers a novel 
procedure for assisting evaluators in the attainment of more reliable procedures than those currently 
in use. Therefore academic leaders charged with the task of evaluating other administrators in 
higher education settings would do well to examine the utility of each of these measures for use on 
their own campus. More importantly, the procedures delineated here offer a promising start for 
future researchers who desire to continue to address this neglected aspect of higher education 
accountability. 

Focusing on administrative evaluation procedures as an additional component of an overall 
program of institutional effectiveness will increase the assessment credibility of academic 
administrators with faculty members. First, administrative evaluation will demonstrate to faculty 
that academic leaders do not fear outcome-based measurement when applied to their own 
employment specific tasks. Secondly, the promotion of administrative evaluation will increase the 
empathy of college administrators for the perspective of faculty who assert,, like Lucus (1996, p. 

239) that “in the final analysis - literally - only so much can be weighed and measured.” Finally, an 
efficient, psychometrically sound, and well-supported program of administrative effectiveness can, 
when used with other components of institutional assessment, contribute to the most desired and 
elusive outcome in higher education, demonstrable improvement in student learning. 
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